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ABSTRACT 

A RAND research team conducted a broad review of the general 
literature on the assessment of quality and productivity in education and 
professional development. The team also reviewed the documentation of 
organizations engaged in such assessment, interviewed experts, attended 
conferences, and conducted site visits to exemplary organizations. This 
report synthesizes the information and provides suggestions for approaches 
that might be useful for agencies given the task of ensuring the quality and 
productivity of education and professional development activities in a 
specific system. "Assessment," as used in this monograph, means the start-to- 
finish process of examining quality or productivity, while "evaluation" is 
the step in the assessment process in which performance measures are 
examined, and a judgment about performance is made on the basis of that 
examination. Regardless of the assessment model selected, the study found 
that three key steps must be included in any provider or student assessment: 
(1) identifying goals of the education activities under consideration; (2) 
measuring the outcomes related to those goals; and (3) evaluating whether the 
outcomes meet those goals. Each step should be linked to the others, and the 
process as whole should be driven by the goals. Six appendixes contain 
supplemental information about the research methodology. (Contains 7 tables, 

1 figure, and 137 references.) (SLD) 
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Executive Summary 



Those responsible for education and professional development within systems 
such as corporations, state governments, and government agencies are con- 
cerned about the quality of those opportunities. As a result, they increasingly 
assign responsibility for ensuring the quality and productivity of education 
within the system to one particular office or agency. Often, such agencies 
receive little guidance about how to approach their task. 

A RAND research team conducted a broad review of the general literature 
on the assessment of quality and productivity in education and professional 
development. The team also reviewed the documentation of organizations 
engaged in such assessment, interviewed experts, attended conferences, and 
conducted site visits to exemplary organizations. This report synthesizes that 
information and provides suggestions for approaches that might be useful for 
agencies given the task of ensuring the quality and productivity of education 
and professional development activities in a specific system. (. Assessment as used 
in this monograph means the start-to-finish process of examining quality or 
productivity, while evaluation is the step in the assessment process in which 
performance measures are examined and a judgment about performance is 
made on the basis of that examination.) 

Why Is System-Level Assessment Needed? 

Although the main task of assessment focuses on the quality and productivity 
of specific providers of education and professional development, the study 
found that a higher-level assessment of the system as a whole is also crucial. 
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Such an assessment has two main purposes: (1) to determine whether the 
stakeholder and system-level needs are being addressed, and (2) to identify 
opportunities to improve efficiency in existing programs. In the first case, 
system-level assessment compares the needs of the population served with the 
programs offered in the system. In a corporate setting, for example, such an 
assessment might find that certain corporate-level goals are not being addressed 
by education and training programs run by individual business units. In higher 
education, a system-level assessment might find that certain geographical 
regions are not being well served by existing institutions in a state. 

To achieve the second aim, the assessment examines whether the systems 
resources are being allocated efficiendy. A number of organizations are improv- 
ing their productivity through this process. 

• The Texas Higher Education Coordinating Board conducts regular pro- 
gram reviews to assess whether a proposed program is based on established 
needs, whether it duplicates other programs in the same area, and whether 
it falls within an institutions mission. 

• At Lucent Technologies, corporate oversight has streamlined education and 
professional development by assessing whether limited education and train- 
ing resources are being used in a way that promotes overall corporate goals. 
The focus on business needs rather than student demand allowed them to 
reduce the number of courses taught throughout the corporation from 
70,000 to approximately 2,000. 

• In the U.S. Air Force, the Air Force Occupational Measurement Squadron 
surveys every person in a particular occupational specialty to identify the 
skills used and not used in particular jobs. Based on this information, Air 
Force managers assess the content of specific training programs to elimi- 
nate irrelevant instruction from courses and ensure that graduates acquire 
the skills and knowledge they need to do their jobs. 

A clear trend in all the systems considered in this study is the development 
of a learning organization of some sort that is responsible for more than just 
the assessment of existing providers. These organizations promote communi- 
cations among stakeholders and develop a clear link between education and 
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professional development on the one hand and the basic mission of the sys- 
tem on the other. Corporate learning organizations describe this relationship 
as “becoming a strategic partner” in the corporation. Such an organization 
facilitates dialogue among key stakeholders, assembles information on work- 
force needs and existing programs, and serves as an interface between cus- 
tomers and providers. 



What Approaches Are Used to Assess Providers 
and Certify Students? 

In reviewing a wide variety of assessment approaches, this study identified key 
similarities and differences among the approaches and classified them into four 
basic models. The first model involves the use of an intermediary organization 
that is responsible for reviewing the process used by individual providers to 
assess their own quality and productivity. In the second model, an intermedi- 
ary organization conducts the actual assessment of providers. In the third 
model, providers conduct their own assessment with no involvement of an 
intermediary. The fourth model differs from the other three in that it focuses 
on the learner rather than the provider and involves the certification of stu- 
dent competencies. Each approach has strengths and weaknesses that make it 
more appropriate for some circumstances than for others. For that reason, no 
one approach can be considered a best practice. The best approach depends 
on the context of the assessment. 



How Does One Choose a Model? 

Many organizations whose job is to ensure the quality and productivity of edu- 
cation and professional development activities can be described as intermedi- 
ary organizations. An intermediary is neither a provider of education and 
professional development nor a direct consumer of the services of such 
providers; it is an entity that promotes communication between the two. Mod- 
els One, Two, and Four, described in this report, allow a role for an interme- 
diary and are therefore the most relevant to such entities. Intermediaries might 
also wish to learn about the best practices under Model Three, however, to 
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serve as a clearinghouse of information useful to provider institutions and to 
remain abreast of new assessment techniques initiated by providers. 

The study identified six factors as the most important to consider in choos- 
ing an approach to assessing the quality and productivity of providers: (1) pur- 
pose of the assessment (accountability versus improvement), (2) level of 
authority, (3) level of resources, (4) centralization of operations, (3) system 
heterogeneity, and (6) system complexity. We argue that Model One is par- 
ticularly well suited to highly complex and decentralized systems. Model One 
is also most suitable for assessors who have little formal authority 
over providers and uncertain resources. This model is based on quality 
improvement concepts that have been used in the business world for the last 
twenty-five years and were adopted by the International Organization for 
Standardization (ISO) in the 1980s to promote high-quality standards among 
manufacturing companies. To qualify for this certification, an organization 
must define and document its quality standards for producing its goods or ser- 
vices in a policy document or quality manual that is reviewed by a third party. 

The academic audit, a new approach to education assessment that has been 
influenced by the ISO, is another example of Model One. The audit is con- 
ducted by an intermediary organization and focuses on ensuring that providers 
of education have effective processes in place for measuring their own quality 
and thus can engage in ongoing self-improvement. Because this approach is 
more sensitive to the different missions and characteristics of institutions than 
are other approaches, it is particularly useful for systems with a diverse set of 
providers. 

The key advantage of Model One is that it delegates to provider organiza- 
tions the task of defining goals, measuring outcomes, and evaluating outcomes. 
As a result, this approach can accommodate a system with many diverse 
providers. Because they have such control over their own assessment, providers 
are less likely to resist the process and are more likely to use it to promote 
improvements. 

The primary disadvantage of Model One relative to Model Two is that it 
emphasizes improvement over accountability. The trade-off between these two 
purposes of assessment remains an important issue for assessors. Model Two is 
better suited for accountability purposes, provided that the intermediary has 
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the authority to ensure compliance. In Model Two, the intermediary sets the 
goals of the assessment, measures performance on these goals, and evaluates 
the performance. This model provides the intermediary with control of the 
assessment process suitable for accountability purposes. In these cases, 
the intermediary can focus on system-level goals, goals that the provider might 
otherwise ignore, to ensure that the provider is meeting the needs of the sys- 
tem. For example, many state legislatures mandate that higher education insti- 
tutions provide data for “report cards” that grade institutions on how well they 
perform on goals such as graduation rates and contribution to the states eco- 
nomic development. The main drawback to Model Two is that any approach 
imposed from an external organization runs the risk of focusing on inappro- 
priate measures and failing to reflect institutional goals. In this case, providers 
may fail to comply with the request for information. Even in cases when 
providers do comply with requests for information, they may not internalize 
them or perform well on the goals set by the intermediaries. Thus, using 
Model Two does not necessarily result in institutional improvement. 

Although Model Three is better suited for improvement, it does not 
include a role for an intermediary. Because Model Three is enacted by 
providers, its use is not constrained by system-level issues of complexity, 
authority, centralization, or heterogeneity. Despite the provider s control pre- 
scribed by Model Three, some of the most innovative examples of this model 
incorporate the perspectives of a range of stakeholders. In these cases, even 
though the provider is responsible for defining, measuring, and evaluating the 
attainment of its own goals, other stakeholders can be involved in these three 
steps. For example, administrators of the Urban Universities Portfolio Project 
use advisory boards comprising business leaders, government representatives, 
and educational experts to advise them on appropriate goals, indicators, and 
measures. In addition, intermediaries such as regional accrediting bodies are 
invited to use the resulting performance information in their assessment 
processes. Therefore, although Model Three is provider initiated, it can evolve 
into a process with a role for intermediaries. 

Model Four represents a completely different approach to assessment, one 
that focuses attention on the learner rather than the provider. This 
competency-based approach can be used in assessment systems for both 



Ensuring Quality and Productivity in Higher Education 



9 



vn 




accountability and improvement purposes and may be similarly immune to 
system-level constraints such as complexity, authority, heterogeneity, and cen- 
tralization. Although Model Four focuses on student competencies, it indi- 
rectly holds institutions accountable by withholding competency status from 
students who have not received the requisite education from specific providers. 
These providers must change to maintain their ability to attract students; in 
this way, the assessment process stimulates improvement while indirectly hold- 
ing providers accountable for change. This approach is very attractive to 
employers and others who want to ascertain whether individuals have specific 
knowledge, skills, or abilities. This assessment can be time-consuming and 
expensive to carry out, however, especially if the competencies are abstract 
ones, such as critical thinking or problem solving. This approach may be there- 
fore best suited for cases in which the knowledge required is easy to ascertain 
and assess, such as in training programs for specific occupations. 



What Is the Three-Step Process of Assessment? 

Regardless of the model selected, the study found that three key steps must be 
included in any provider or student assessment: 

• Identifying goals of the education activities under consideration 

• Measuring the outcomes related to those goals 

• Evaluating whether the outcomes meet those goals. 

The literature review revealed several broad lessons concerning these steps. 
First, each step should be linked to the others, and the process as a whole 
should be driven by the goals. It is especially important to avoid selecting mea- 
sures before or without defining goals. Practitioners in higher education, cor- 
porate, and government settings stress the tendency of people to value what 
is measured and focus exclusively on that information rather than linking 
what is measured to the purpose of the activity. 

Second, developing measures that relate to goals is a crucial if difficult step. 
It is often difficult to find an adequate measure of achievement for a particu- 
lar goal. It is usually better to use an imperfect measure of a specific goal than 
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it is to use a perfect measure of something different, however. Engaging a 
broad range of stakeholders in this process helps to keep it focused on the goals 
of the undertaking. Such stakeholder involvement and continuous feedback 
is an explicit element of both the Baldrige Award process and the balanced 
scorecard. 

Third, the trend in assessment is to focus less on input measures and more 
on process and outcome measures. Measuring outcomes alone may not 
result in improvement, but considering the intervening processes that use 
resources to produce outcomes provides information more useful to program 
improvement. 

Finally, except for certificate or licensing programs, providers of profes- 
sional development courses are not likely to be able to rely on preexisting eval- 
uation tools with known validity and reliability characteristics. Rather, they 
will most likely have to develop measures of learning outcomes on their own. 
The literature provides some guidelines for developing such measures and for 
avoiding major sources of invalidity and unreliability. Intermediaries can play 
an important role by applying these guidelines to their own assessment 
processes and acting as clearinghouses of such information for providers 
engaged in assessment. 
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Foreword 



Each week the Chronicle of Higher Education is filled with stories about legis- 
lators and the public s concern about the cost and quality of higher education 
and the need for the system to respond to more problems of greater com- 
plexity, such as the need to resolve international tensions around globaliza- 
tion. In response to these calls for change, many new systems — report cards, 
a national study of student engagement, assessment, institutional research, and 
the like — purport to aid higher education in examining its structures 
and processes. In the last decade, leaders in higher education have come to 
realize that assessment or some new forms of accountability and improvement 
are likely to become part of the enterprise. Faculty, administrators, and even 
students are aware of the pressures to be accountable and to improve higher 
education. But how do we make sense of all these newly developed mecha- 
nisms? Which ones work and how do they work? Although this monograph 
does not review all these systems, it provides a framework to examine and eval- 
uate these various approaches to accountability and improvement. 

The ASFIE-ERIC Higher Education Reports have long been committed 
to understanding and presenting the best research on assessment. This 
monograph emerged out of an extensive national project conducted by 
RAND. It is another expression of this long-standing commitment. Susan 
Gates, Catherine Augustine, and Tessa Kaganoff provide guidance on system- 
level assessment, which will be invaluable to legislators, trustees, governing 
boards, state systems, accreditors, and other individuals and groups entrusted 
to ensure the vitality of higher education. The monograph is also an impor- 
tant contribution because it examines our assumptions about assessment. Most 
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assessment efforts focus on accountability and pay only lip service to improve- 
ment. Is society willing to spend the time and money to develop extensive 
assessment practices that do not run deeper than merely understanding 
whether higher education is meeting its commitment to society? It seems crit- 
ical to focus on how assessment can be used for improvement and to design 
systems that are effective in meeting this goal. I applaud the authors for not 
only asking the hard questions (such as why goals and methods are often 
unaligned) but also developing constructive solutions. It is easy to say system- 
level assessment does not work as well as it should, but it is difficult to develop 
practical and implementable ideas for improving it. 

The monograph includes a focused literature review as well as a discussion 
of research results from a national study. Although the literature synthesizes 
concepts about system-level assessment, it has implications for assessment at 
all levels, especially around issues of alignment of goals and design. The 
authors’ work uncovers important principles for advancing our assessment 
practices, such as the need to more clearly identify the goals of the process up 
front, choose appropriate measures, and use those measures to evaluate 
progress toward those goals. 

Several other ASHE-ERIC Reports are important supplements for this 
monograph. Gaither, Nedweck, and Neal, in their monograph Measuring Up , 
focus specifically on performance indicators as a method of accountability 
among state systems. Alstete reviews approaches to benchmarking, a particu- 
lar method of assessment used by many campuses for improvement across the 
institution in programs, departments, schools, and colleges, whereas Creamer 
examines assessment for a particular group (faculty) in her monograph Assess- 
ing Faculty Publication Productivity. Each monograph reviews different aspects 
of assessment that can be used for advancing campus efforts related to 
improvement and accountability. 



Adrianna J. Kezar 

ASHE-ERIC Series Editor 
University of Maryland 
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Introduction 



R EADERS OF THE higher education literature and practitioners in the 
field are aware of the web of accountability arrangements in which col- 
leges and universities exist today. Various constituents, including students, par- 
ents, government officials, and employers, have suggested one method or 
another for holding higher education institutions accountable (Banta, 1988; 
Banta and Borden, 1994; Boyer, Ewell, Finney, and Mingle, 1987; Council 
for Higher Education Accreditation, 2000; Ruppert, 1995; Schulz, 1996). The 
end result of these demands is a complex array of assessors, including accred- 
iting agencies, state higher education boards, professional societies, and indi- 
vidual customers (Albright, 1995; Ewell, 1987b; Ewell, 1993; Ewell, 1999b; 
Lenth, 1996; Stevens and Hamlett, 1983). This phenomenon is not unique 
to higher education; demands for accountability have also increased in the fed- 
eral government. In the 1990s, Congress passed several pieces of legislation, 
including the Government Performance and Results Act (GPRA) of 1993, 
that address waste and inefficiency, increase program effectiveness, and 
improve the internal management of the federal government. The GPRA 
directs the 24 largest federal agencies to submit five-year strategic plans as well 
as annual performance plans with their budget requests to Congress (Office 
of Management and Budget, 1998). 

Although researchers have devoted substantial attention to how providers 
of postsecondary education respond to these calls for accountability and how 
the system might change as a result, little attention has been devoted to under- 
standing the role of those entities responsible for ensuring accountability. What 
do they do? How is what they do different from the role of other assessors? 
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